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Reform Support Network 


Introduction 


A Toolkit for Implementing 
High-Quality Student 
Learning Objectives 


The Race to the Top Reform Support Network (RSN) 
designed this toolkit to help States and school districts 
implement student learning objectives (SLOs) with the 
highest degree of quality. These SLOs have the 
following characteristics: 


SLOs set high yet attainable expectations for 
students and teachers. 


Success on SLOs correlates with success on other 
high-quality measures in the teacher evaluation 
framework. 


SLOs support improved instruction and learning. 


Educators and evaluators perceive SLOs as 
valuable in strengthening teacher practice and 
fostering a meaningful professional dialogue 
about student learning. 


The toolkit has three major objectives: 


Provide information on SLO design and 
implementation approaches to support States’ 
decisions about how to ensure high-quality, 
sustainable implementation of SLOs 


Provide easy access to tools and resources that 
States can learn from and adapt 


Promote interaction and custom viewing of toolkit 
content based on each users’ needs 


The toolkit enhances the first RSN Quality Control 
Toolkit, published in the winter of 2012 and produced 
by the Student Learning Objective Work Group. As 
part of their participation in the Race to the Top 

grant program, leaders from 10 State departments 

of education joined the work group and developed, 
among other activities, a framework for quality control 
that the RSN has revised in collaboration with the now 
12 States in the SLO Work Group. 


This toolkit is divided into the five components of 

the framework and highlights activities and tools 
critical for a quality SLO system. It contains templates, 
guidance documents and other artifacts from the field 
that States and school districts can select or adapt. 
Some of the components have more tools than others. 
Given the current state of the art in certain areas of 
SLO implementation, other components are less 
robust. 


Figure 1. A Framework for Creating High-Quality SLOs 


1. Provide tools for 
developing, approving 
and scoring quality SLOs 


5. Ensure 
continuous 
improvement 


2. Foster selection 
of quality assessments 
and rigorous targets 


3. Communicate 
with teachers 
and principals 


4. Train district 
staff and school 
administrators 


The RSN is grateful to the innovators in State and 

local education agencies who produced the materials 
included in this toolkit. They provide a foundation 
upon which States can build a quality SLO system that 
allows teachers of non-tested grades and subjects 

to participate with rigor in new pay and evaluation 
systems that require measures of student growth. 
While the tools herein represent the state of the art, 
State Education Agencies (SEAs) and local education 
agencies (LEAs) will produce improved tools over time. 
In the spirit of continuous improvement, the RSN will 
update the toolkit as the field gains greater insight and 
develops even stronger approaches. 


Reform Support Network 


SLO Context: Definition, Benefits, 
Challenges and Policy Choices 


SLO Definition 


SLOs are content- and grade/course-specific learning 
objectives that educators can validly measure to 
document student learning over a defined and 
significant period of time.' They have become the 
most common method for measuring student 
learning and growth for teachers of non-tested grades 
and subjects in evaluation and compensation systems. 
Some States also require teachers of tested grades 

and subjects to develop student learning objectives 
for evaluation purposes. Although specifics of the SLO 
approach vary across jurisdictions, the implementation 
process is similar in most, though not all, States and 
school districts. At the start of the school year, the 
teacher, principal or district examines baseline data, 
sets one or more learning goals for the teacher's 
students and identifies a measure or measures for 
assessing progress towards them. In most cases, the 
principal, an evaluator or the district approves the 
learning goals and the assessments. Teachers later sit 
down with their evaluators for a mid-term conference 
to discuss student progress toward the learning goals 
and to adjust strategies if students are not progressing 
as expected. At the end of the year, the teachers and 
evaluator assess student progress, and the evaluator 
determines how well the teachers succeeded in 
getting their students to meet the learning goal(s). This 
determination leads to a score that factors into each 
teacher's evaluation. 


SLO Benefits 


This process of setting goals, monitoring progress 
against those goals and evaluating performance is 
strong instructional practice. Effective teachers gather 
data about their students, set goals based on that 
information and then assess whether the goals have 
been met. SLOs promote these strong practices across 
schools and districts, and offer additional benefits: 


' Marion, S.F., DePascale, C., Domaleski, C., Gong, B., and Diaz- 
Bilello, E. (2012). Considerations for Analyzing Educators’ 
Contributions to Student Learning in Non-Tested Subjects and 
Grades with a Focus on Student Learning Objectives. 


« SLOs promote reflective and collaborative 
teaching practices. Rich discussions about 
student learning that occur during and after SLO 
development can improve instruction. These 
discussions happen between teachers and their 
evaluators and among teachers in a grade level or 
subject area. 


* Teachers perceive SLOs as relevant and 
empowering as compared with school-wide, 
value-added measures. The SLO process allows 
principals and teachers to influence how teachers 
are evaluated and design learning objectives for 
each teacher's course and students. 


+ SLOs promote aligned curriculum, assessment 
and standards. The SLO process typically requires 
teachers and principals to identify the standards of 
focus and to map how assessments will measure 
progress against those standards. In designing an 
SLO, teachers must consider how their curricula and 
instructional strategies will help students meet the 
standards. 


+ SLOs are adaptable to any teacher. Teachers of all 
grades and subjects can demonstrate their impact 
on student learning with SLOs, because SLOs do 
not rely on standardized assessments. 


SLO Challenges 


Unlike value-added measures that are standardized 
and statistically based, teachers write SLOs in most 
jurisdictions, and they may use different assessments 
and different growth targets depending on where 
their students are starting academically. Because 

of this variability, States and school districts face 

the challenges of ensuring the quality, rigor and 
comparability of SLOs across classrooms, districts and 
entire States. 


Yet, States and school districts cannot expect their 
SLOs to yield the same scientific validity and reliability 
that value-added measures based on high-quality, 
standardized State assessments produce. That is simply 
not possible. Nevertheless, there is strong precedent 

in other fields for using goal setting in a consistent, 
credible manner. Employers and employees in many 


Figure 2. SLO-Process Flexibility Spectrum’ 
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American industries sit down together annually to set 
objectives and identify the metrics they will use to 
determine whether they have been met. Employers 
make decisions about their employees—whether 
to sign them up for training or to promote them, for 
instance—based on the results of the objectives. 
And they do so without using psychometric 
methods to prove that the metrics are relevant, or 
that expectations have been met. Still, employees, 
including teachers, should expect a fair, rigorous 
and high-quality process of setting objectives and 
implementing them. 


The RSN has designed this toolkit to help States and 
school districts meet this challenge. 


A Spectrum of SLO Approaches 


As State policy and district innovation create SLO 
systems across the country, a distinct range of SLO 
policies and rules have emerged. These policies differ 
in the degree of standardization or flexibility they grant 
teachers and evaluators when they create and score 
SLOs. There is typically a tradeoff between flexibility 
granted and comparability of SLOs across schools and 


? Analysis of the American Institutes for Research (page 13) and 
the experience and analysis of Education First informed this 
graphic. 


Minimum Flexibility 


More Comparable SLOs 


districts. The more flexibility principals and teachers 
have to set learning targets, select assessments and 
determine the approval and scoring process, the less 
comparable the SLOs. 


Some jurisdictions (such as Rhode Island, Colorado 
and New Jersey) prefer SLO systems with more 
flexibility in target setting, assessment and scoring. 
These States value goal setting based on discussions 
about teaching and learning between teachers and 
evaluators and on the needs of each teacher and 
student. Their reliance on the judgments of teachers 
and evaluators results in more target, assessment and 
scoring variations than systems that require greater 
standardization. Some jurisdictions that aspire to 
flexibility launch their SLO work on the standardized 
side of the spectrum with the intention of granting 
more flexibility over time, because they do not feel 
confident in the capacity of their schools to implement 
SLOs at the outset. 


Other jurisdictions (such as Georgia and New York) 
promote SLO systems that take greater control of 
target setting, assessments and scoring. Doing so, 
they believe, results in a greater level of comparability 
across classrooms. Teachers who have a similar 
impact on student growth should receive similar 


Figure 3. SLO-Process Flexibility Spectrum, by State 
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growth scores. These States want to promote parity 

in expectations and credibility between teachers of 
non-tested grades and subjects and teachers of tested 
grades and subjects who receive value-added or other 
standardized growth measures. Some jurisdictions 
aspire to standardization, but because they lack high- 
quality common assessments, they launch their SLO 
work on the flexible side of the spectrum with the 
intention of standardizing over time. 


States should determine what they would like to 
accomplish through their SLO systems and then 
decide the amount of flexibility they want to allow 

in the SLO process to accomplish their goals. If a 

State’s major priority is to have teachers improve their 
practice and deepen their dialogue with administrators 
about student learning, it will likely veer more toward 
SLO policies that grant flexibility. If its foremost priority 
is to create as valid and reliable a measure as possible 


Figure 4. SLO-Process Flexibility Spectrum, by Policy Topic 
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Teachers select from Common assessment 
a prescribed list of used for teachers of the 
assessments same grade or subject 
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District or State provides 
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content and minimum 
and maximum target 


District or State dictates 
priority learning content 
and targets 
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District State 


SLO Learning 
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Content and j 
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Targets 
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SLO Approval Evaluator 
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evidence 


Some SLOs with Pre-set quantitative 
standardized scoring, bands guide scoring or 
others with evaluator- scoring plan included in 

judged scoring SLOs 


to compare teacher impact on student learning, then 
a State will favor policies that promote standardized 
SLOs. These priorities are ultimately not mutually 
exclusive. In a more flexible system, with the right 
training, tools and monitoring, teachers and principals 
over time will improve the quality of their SLOs and 
thus the consistency, credibility and comparability 

of the SLO measure. In a standardized system, States 
and school districts can build practices into the SLO 
process that generate student-learning dialog among 
educators. But States have to choose a starting priority 
around which to organize their policies. A host of 
policy choices cascade from this first positioning, some 
of which are outlined below. 


A State's approach to SLO policy has significant 
implications for the locus and degree of SLO 
implementation and quality control. States with 
policies that promote SLO standardization will have a 
higher implementation lift at the district level, whereas 
States that promote flexibility will have a higher 
implementation lift at the school level. The more 
standardized the SLO process, the less work teachers 
have to do to draft their SLOs and the less reliant the 
process is on evaluators to assure quality and rigor. 
Standardized SLOs require districts to deploy district- 
wide assessments, prescribe how goals will be set 
using those assessments and provide clear guidance 
about what level of performance on the assessment 
will yield a particular score. In contrast, flexible SLOs 
demand a significant, sophisticated participation by 
teachers and their evaluators. This engagement with 
SLOs can yield substantial benefits, but it also requires 
time for training, SLO writing and ongoing monitoring 
for quality. 


Connections with College- and 
Career-Ready Standards 


States are implementing new evaluation systems 
simultaneously with the rollout of college- and career- 


ready standards and aligned standardized assessments. 


Both evaluation reform and new standards are seismic 
changes for the field, and they will be successful 
only if they meaningfully complement and reinforce 


each other. College- and career-ready standards raise 
expectations for students, and the new evaluation 
systems promote new teaching practices that help 
students meet these new expectations. Our challenge 
is to both communicate this intersection and build 

it intentionally into our policies and new practices 

so that educators experience it. An Aspen Institute 
report well describes this challenge and the RSN’s 
Transitions Work Group will soon release guidance on 
how to integrate these reforms further at both the 
SEA and building levels. An evaluation system with 
multiple measures (student growth on State tests, 
classroom observations and student surveys) can 
accurately predict a teacher's effectiveness. 


n the rush to roll out evaluation and standards reform 
at the same time, one common approach has been to 
divide and conquer. State and district administrations 
generally assign different leaders and staffs to the 
initiatives. Although they intend to work together, 
nthe press of implementation, they may neglect 
potential intersections between new evaluations and 
standards. Disconnecting the SLO work from the new 
standards threatens the quality of SLOs because the 
SLOs are less likely to align to the new standards and 
to draw on standards-aligned assessments. Several 
States, however, are working to overcome their 
organizational silos and improve the quality of SLOs 
by connecting their development with the rollout 

of college- and career-ready standards. For example, 
Ohio includes a connection to Common Core State 
Standards in its SLO quality checklist. Georgia has an 
assessment item bank in which each item links to a 
specific standard. Louisiana has a number of annotated 
exemplar SLOs that explicitly link to the Common Core 
State Standards. Colorado has provided for district and 
school leadership implementation timelines and work 
plans that include the Colorado Academic Standards 
(which include the Common Core State Standards), 
assessment transition, improvement planning and 
evaluation-implementation benchmarks. The timelines 
and work plans cover three-month phases and include 
guiding questions and resources. 


Component 1: Providing Tools for Developing, Approving 


and Scoring SLOs 


The SLO process for teachers is similar in most States assess student progress, and the evaluators determine 
and school districts. At the start of the school year, how well the teachers succeeded in achieving their 
the teacher, principal or district examines baseline learning goals. This determination leads to a score 
data, sets one or more learning goals for the teacher's that factors into the teacher's evaluation. Each step in 
students and identifies a measure or measures for the process requires thoughtful communication and 
assessing progress towards them. In most cases, the collaboration between teachers and their evaluators, a 
principal, an evaluator or the district approves the solid understanding of the expectations for SLO quality 
learning goal(s) and the assessment(s). Teachers later and robust tools to support implementation. 


sit down with their own evaluator for a mid-term 
conference to discuss progress students are making 
toward the learning goal(s) and to adjust strategies 


This section of the toolkit provides sam 
to this process of developing, approvin 


ple approaches 
g and scoring 


as well as overall guidance to those wh 


if students are not progressing as expected. At the a 


end of the year, the teachers and their evaluators 


Figure 5. The SLO Process 


o implement 


REVIEWING STUDENT DATA DEVELOPING SLOs APPROVING SLOs 


Individual teachers or teams of teachers Individual teachers or Evaluators review 


and approve 


review student data before the school teams of teachers draft SLOs. proposed SLOs (or request revisions). 


year begins (or they review pre-test data (September-October) (October-November) 
after the school year begins). 


(August-September) 


REVIEWING SLO PROGRESS 


Teachers or evaluators may review teacher 
progress on SLO targets mid-year. Changes 
in assignment or class composition may 
warrant target adjustment. 


Evaluators review and score the progress 
on SLOs by individual teachers or 
teams of teachers. 


(May-June) 


measure or among 


(February) 


REFLECTION 


Individual teachers or teams 


of teachers and evaluators discuss 
progress and implications for 
next year’s SLOs. 


(June, Summer) 


SCORING SLOs SUMMATIVE SCORING 


SLO results are included as the 


the measures 


of student growth. 
(June) 


Developing SLOs 
SLO Templates 


Teachers and administrators use SLO templates to 
create and record their SLOs for a defined interval 

of instruction. SLO templates, which are forms 

that prescribe the information that teachers and 
administrators must provide to create their SLOs, 
scaffold the development of SLOs. Templates walk the 
SLO developer step by step through setting objectives, 
and often call for rigorous reflection and analysis of 
each element. 


State and district templates vary in the content 

they require. Content common to most State and 
district templates includes the student population, 
interval of instruction, learning content, baseline data, 
assessments and targets. Not all States group this 
content in the same way. Rhode Island, for example, 
requires a specific “Objective Statement” element that 
identifies the priority content and expected learning. 
The Denver Public Schools and others have templates 
that require teachers to list the instructional strategies 
they will use to meet their SLO target, and many States, 
such as Ohio, require a rationale for the target. Some 
States incorporate the final SLO scoring mechanics, 

so teachers and evaluators know from the start what 
results they will have to produce to receive a specific 
rating. For example, New York requires all teachers 
and principals to use a State-developed template 

that includes the State's effectiveness-scoring scale, 

so teachers know as they begin SLO implementation 
how evaluators will determine the range of student 


10 


performance that will define their SLO rating. 

Louisiana includes a scoring plan in its template, so 
that teachers and evaluators establish at the outset 
how a rating will be assigned at the end of the year. 


State-provided support for developing SLOs does 
not begin and end with providing templates. Most 
States provide guidance on what makes an element 
of high quality. Rhode Island has created a two- 
page brief that lists characteristics of a strong SLO 
element, and Louisiana (page 13) provides similar 
guidance in its SLO guidance manual. Pennsylvania 
has developed an SLO Help Desk table that presents a 
description, format clarification and example for each 
part of its SLO template. States can also structure their 
templates to promote a comprehensive and high- 
quality implementation. Indiana (pages 36-46) built 
its template around a five-step process that begins 
with the selection of an assessment—noteworthy 

as a Starting point in and of itself-—and concludes 
with an end-of-course judgment about the teacher's 
performance. Although many SLO systems begin 
quality control when evaluators apply rubrics or 
checklists to gauge the quality of an SLO, Indiana 
builds a system of quality control into the template 
itself. 


States can adjust templates in response to teacher 
feedback, as necessary. Rhode Island, for example, 
recently streamlined its template by removing a 
section that teachers found confusing, reordered the 
template’s element order and collapsed several similar 
elements into one. 


Table 1. Common SLO Elements and Guidance for High-Quality Content’ 


Elements Common To State 
and District SLO Templates (@oantantelan (-vant=niaClelcer-lara=) 


Student Population 


Includes the exact number of students addressed by the SLO 


Can describe student population characteristics, including grade, specific abilities, 
needs and any special population status (some jurisdictions request this information 


for baseline data) 


Interval of Instruction 


Teachers can attach student roster with student achievement data as appropriate 


The objective applies to an instructional period, such as an academic year or semester 


Specifies beginning and end dates to clarify when instruction will begin and end for 


the SLO 


Learning Content 


dentifies the standard(s) the SLO will address 


dentifies a subset of the knowledge and essential skills that students are expected to 
master in a particular course 


Describes course curriculum and focus 


Baseline + Includes pre-assessment baseline data and additional data as available, such as 


Assessments 


historical information about student abilities, learning needs or attendance history 
Baseline data can include past performance of similar groups of past students 


easures the standards set forth in the learning content 


Requires students to demonstrate a high level of cognitive processing, including 
igher-order thinking such as analysis, evaluation and synthesis 


h 
Assessments provide meaningful “stretch,’ that is, they allow all students, including 
high- and low-achievers, to demonstrate their knowledge 


Targets - Identifies either a mastery or growth target 


Ambitious yet attainable for the interval of instruction 


Identifies the proper scope of an objective, that is, broad enough that it captures the 
major content and narrow enough that it can be measured 


The target may be tiered so as to be both rigorous and attainable for all students 


included in the SLO 


Approving SLOs 


Once written, an SLO usually goes through an approval 
process informed by a rubric or checklist created by 
the State or district and designed to measure the 
quality of the SLO and determine if it meets approval 
standards. Approval typically occurs at the school 
level, but can also occur at the district or State level. 
High-quality rubrics and checklists are most important 
for those States that emphasize flexibility in their SLO 
systems so that principals and other evaluators have 

a well-calibrated tool that facilitates consistent and 
rigorous SLO approval. 


Approval Rubrics 


Approval rubrics classify the quality of SLOs in three to 
four performance levels (for example, “unsatisfactory; 
“needs improvement,”acceptable,"“excellent”) that 
evaluators use to rate SLOs in each of several domains, 
usually the components of an SLO established by State 
or district templates/frameworks, Helpful examples 

of these rubrics come from Denver and Austin. In 


Denver, teachers and principals rate SLOs in several 


> Targeting Growth Using Student Learning Objectives as a Mea- 
sure of Educator Effectiveness and Student Learning Objectives 
Indicators of a Strong SLO 
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areas: rationale, population, interval of instruction, 
assessment, expected gain, learning content and 
strategies. Some practitioners consider extensive 
rubrics like Denver's to be teaching tools for those 
learning how to write and score SLOs. The learners 
reflect on their ratings across all the elements and 
draw conclusions about how they can write better 
SLOs in the future. 


Approval Checklists 


As more jurisdictions are implementing SLOs, gaining 
experience and learning from their peers, they have 
streamlined tools designed to evaluate SLO quality. 
For instance, Rhode Island simply asks teachers 

and administrators to consider whether an SLO is 
acceptable or unacceptable in three different areas. 
The State developed a quality review tool to help 
educators calibrate their understanding of SLO quality, 
which is aligned to a do-it-yourself SLO audit released 
in the fall of 2013. The development of more simplified 
tools for SLO approval is in fact a trend, with Indiana, 
Ohio and Louisiana (page 13) using tools that 
function not as rubrics but as short checklists aligned 
to criteria for high-quality SLOs. The RSN itself, through 
the leadership of Georgia, Maryland, New York and 
Rhode Island, has developed a draft approval checklist 
tool. 


Reviewing SLO Progress 


Many States suggest that teachers and evaluators 
conduct a midcourse check-in to monitor how 
students are progressing toward SLO targets. These 
conversations focus teachers and evaluators on 
student learning and the instructional practices that 
will ensure students meet their goals. This high-quality 
professional development practice demonstrates to 
teachers and administrators alike the power of SLOs to 
improve instruction. Indiana and Arizona (page 39) 
provide practitioners with midcourse check-in 
templates to guide this discussion and promote 
actionable feedback. Some jurisdictions permit 
evaluators to adjust SLO targets at the midcourse 
check-in if the teacher's students or assignment have 
changed substantially. 
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Scoring SLOs 


District evaluators use scoring guidance and rubrics to 
measure how teachers have performed in pursuit of 
their SLO targets. Scoring criteria are typically locally 
controlled, with schools or districts defining success— 
though many simply implement the guidance 
provided by States. 


Delineating Performance Levels 


Almost all districts and States differentiate teacher 
performance on SLOs by establishing four or five 
performance levels, which generally correspond to 
substantially missing the target, nearly hitting the 
target, hitting the target or exceeding the target. 
Rhode Island (page 19) uses the terms “minimal,” 
“oartial,“full” or “exceptional” attainment. Other 
jurisdictions, such as New York (page 16), describe 
success in aligning with its evaluation framework’s 
effectiveness levels (“highly effective;"“effective/’ 
“developing’ or “ineffective’—a sequence otherwise 
known as “HEDI"). The SLO performance levels correlate 
to a numeric score that is included in the teacher's 
summative evaluation rating. 


Calculating Performance Levels 


States and school districts must decide how much 
evaluator judgment plays a role in determining the 
performance level an SLO earns. Evaluator discretion 
in scoring results is less reliable scoring but may 
foster more teacher and evaluator dialogue and allow 
them to take into account on-the-ground realities. 
On the other hand, a more standardized approach to 
scoring that allows for less evaluator discretion can 
maximize the predictability of the SLO score. Three 
different approaches to performance-level calculation 
showcase a State's range of options. 


Standardized Approach 


Some States standardize their SLO scoring process by 
indicating which student performance outcomes will 
yield which SLO score. Ohio, for example, recommends 
that districts use an SLO scoring template to input 
student baseline scores, individual growth targets 

and post-test scores to calculate the percentage of 


students that meet or exceed their growth target. 
Teachers and evaluators then reference a scoring 
matrix that reveals the SLO rating associated with one 
of five possible percent ranges. 


Similarly, Indiana (pages 26-28) requires teachers 

and evaluators to set mastery goals for whole classes 
of students and then measure teacher effectiveness 
against the percentage of students who perform 
within a specific numeric range. For instance, a scoring 
example in an Indiana SLO guidebook (page 19) 
labels a world history teacher highly effective if at least 
90 percent of her students achieve 85/100 or better 
on the end-of-course world history assessment. For 

an effective teacher, that percentage is 74 percent. 

For an ineffective teacher it is below 54 percent. Many 
States adhere to an approach that utilizes percentages, 
including Louisiana (pages 16-17) and Georgia. 


Flexible Approach 


Others take a more flexible approach that allows 
evaluators to examine the performance of teachers 
against their objectives more holistically. Rhode 

Island (pages 19-21) and Maryland allow teachers 
and evaluators to use multiple measures to determine 
if objectives have been met, requiring evaluators to 
examine a body of evidence and make determinations 
based on a reasoned review informed by a scoring 
process map. 


Hybrid Approach 


Some States take aspects of both approaches. While 
Indiana (pages 26-28) requires strict reliance on 
numeric measures for whole-class objectives, it has 
less strict requirements for SLOs written for targeted 
populations. In fact, for target-population SLOs in 
Indiana, “evaluators decide which performance level 
best describes the effect the teacher had on his or 
her students’ learning,’ the Indiana SLO guidebook 
RISE suggests, “This decision requires professional 
judgment’ 


New York draws on both approaches as well. While 
all teachers and principals must translate their scores 
into HEDI levels, SLOs in courses that use a State 


assessment must have an objective based on State 
averages for similar students, and the HEDI levels are 
predetermined. For other teachers and principals, New 
York gives the district flexibility to determine how to 
score SLOs and align them to the HEDI scale, though 
the State requires educators with multiple SLOs to 
weight the results of the SLOs proportionately based 
on the numbers of students in each SLO. 


Using Technology to Power 
the SLO Process 


Many jurisdictions are investing in technology 
platforms to capture valuable data and support 
educators as they complete each step in the SLO 
process. Rhode Island has implemented a statewide 
system to support the entire evaluation process, 
from professional growth plans and observations 

to approving and scoring SLOs, including capturing 
teachers final evaluation ratings. Ohio has 
implemented a similar statewide system for capturing 
SLO ratings, and Pennsylvania has a platform that 
allows teachers to record SLOs in an online template 
that they can share with principals to gain electronic 
feedback. Washington, DC Public Schools (DCPS) 

and Denver Public Schools have developed more 
comprehensive Web-based systems for managing 
and tracking SLOs. These systems streamline the SLO 
process by allowing educators to complete the entire 
SLO process online, including submitting, approving 
and scoring SLOs at the appropriate time during the 
school year. DCPS even pre-populates a target for 
each teacher based on his or her grade, subject and 
the district-wide guidance on target setting. Teachers 
can refine and customize the targets if they choose, 
but the system always provides an appropriate default 
target that teachers can use as a Starting point. 

These systems support quality control by promoting 
good practice, making the SLO process manageable 
and supplying data for monitoring and continuous 
improvement. 
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Component 2: Fostering the Selection of Quality 


Assessments and Targets 


Assessment selection and target setting are the crux of 
the SLO process. Assessments are the primary source 
of data educators use to gauge student performance. 
Poorly designed assessments do not accurately 
measure student abilities. Therefore, if SLOs are based 
on low-quality assessments, the SLO process cannot 
yield accurate or meaningful results. But even when 
assessments are of high quality, if teachers set easily 
attainable or overly ambitious targets for student 
performance on assessments, the SLOs will not be 

a credible measure of teacher impact on student 
growth. 


Strategies for Promoting 
Assessment Quality 


An assessment can be any measure that allows 
students to effectively demonstrate what they know 
and can do, such as a performance task, portfolio or 
standardized test. States and school districts must 
subject assessments used for SLOs to a rigorous 
quality-assurance process, while considering time (to 
create, administer and score), cost, available expertise, 
resources and quality. Those who use assessments 
therefore need guidance, as they either procure 
existing assessments or develop new assessments to 
support SLO implementation. 


Some States allow flexibility in assessment selection for 
SLOs, while other States are more prescriptive. States 
with more flexibility give educators great latitude to 
choose or create an assessment for their SLOs. These 
States provide guidance and training for districts and 
teachers on the selection of high-quality assessments, 
and some States provide assessment banks of pre- 
screened assessments from which educators can 
draw. On the other end of the spectrum, States and 
districts with more standardization either mandate 
specific assessments for the various non-tested grades 
and subjects or they determine a list of pre-approved 
assessments from which teachers and districts must 
choose. 
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Guidance and Training for Selecting or 
Developing High-Quality Assessments 


State guidance for assessment development 
commonly asks implementers of SLOs to examine the 
quality of assessments chosen to measure student 
learning, the assessments’ alignment to district or State 
standards and the procedures used in administering 
them. Rhode Island provides a worksheet that 
teachers or LEAs can complete to justify the use of 

an existing or teacher-created assessment. Georgia 
(pages 39, 51-57) requires its districts to apply a 

table of specifications and a measure-criteria table 

to assessments they create for SLOs (districts use the 
same devices to select pre-existing measures). Ohio 
provides detailed guidance on selecting a high-quality 
assessment, including examples for a variety of grades 
and subjects and a summary check list. Colorado 
provides a comprehensive tool for reviewing, editing 
and revising locally created assessments designed 

to help educators rate an assessment’s potential for 
measuring student academic growth aligned to the 
Colorado Academic Standards. 


Providing guidance on assessment selection and 
development is important and helpful but not entirely 
sufficient. Educators need the knowledge and skills 
to choose and create assessments, which typically 
requires States and districts to help teachers develop 
these skills. Many States have invested in assessment 
literacy training and support for districts. In Georgia's 
first year of SLO implementation, the State provided 
four days of assessment development training for 
teachers and content specialists in various regions of 
the State. The trainings focused on items, rather than 
complete assessments, in order to create a sense of 
ownership of assessment design. Rhode Island offers 
two interactive modules on its Website: Assessment 
Literacy and Using Baseline Data to Set Targets. 
Rhode Island also recently launched an Assessment 
Toolkit with four distinct tools for developing and 
selecting quality assessments, using baseline data, 


reviewing assessments and collaborative scoring. Ohio 
has developed a three-step training process, which 
includes delivery of 100 one-day trainings for district 
and building teams, a series of six online modules 
that will be available through the SEA’s Website and 
six assessment-literacy specialists to help expand 
professional development (PD) at the district level. 


Assessment Banks 


Many districts struggle with similar gaps in 
assessment coverage for a large number of grades 
and subjects. Some States are therefore coordinating 
the development of statewide assessment banks to 
address the gaps. Colorado organized and supported 
statewide collaboratives of teachers of common 
subjects, and asked them to identify and create high- 
quality assessments that are fair, valid and reliable 
measures of student learning. The State catalogs the 
assessments in a resource bank and makes them 
available to all districts. The Colorado Department of 
Education Resource Bank includes assessments for all 
grades and content areas. They provide districts with 
assessment options or starting points from which they 
can build their own assessments for student learning. 
Districts may also use the assessments to measure 
teacher effectiveness. Georgia also coordinated 

the development of assessments by inviting teams 
of educators across the State to work together. 
Throughout the assessment development process, 
SEA staff trained and supported the teams. As in 
Colorado, Georgia permits districts to use the vetted 
public domain measures, customize them or use their 
own locally developed or procured measures. 


Identifying Pre-Approved Assessments 


Louisiana provides a list and rank of assessments 

by tier, identifying tier-one assessments as those 
produced by the Bayou State, another State or 
companies that develop assessments for national 
consumption; tier two as credible assessments aligned 
to State standards; and tier three as teacher-developed 
assessments or those that use indirect measures of 
student learning. Louisiana recommends that teachers 
use tier-one assessments when available. 


New York also provides a list of pre-approved, third- 
party assessments. The State requires districts that 
wish to use a third-party assessment to choose one 


from the list for applicable teachers or principals. 
New York further allows districts, regions and Boards 
of Cooperative Educational Services (BOCES) to 
develop their own assessments for the purposes of 
SLOs. However, the superintendent must ensure that 
the assessment is rigorous and comparable across 
classrooms, in accordance with the commissioner's 
regulations. 


Setting Rigorous and 
Appropriate Targets 


Target Setting with Baseline Data 


Setting the right targets or objectives is one of the 
most challenging parts of the SLO process. Targets 
should identify the expected learning or mastery 
outcome for the defined instructional period. Some 
States and school districts provide specific guidance 
on how to set targets, some tell teachers what their 
target should be and others allow greater latitude. 
The challenge in setting targets is that teachers and 
administrators must give careful thought to whether 
they have set an objective that is rigorous—yet 
realistic. 


Targets are intended to convey growth and are 
typically framed as a percentage of students meeting 
a certain score or pass rate on an assessment (mastery 
target) or as a percentage of students demonstrating 
a certain amount of growth on an assessment (growth 
target). For example, Ohio (pages 9-10) gives the 
following exemplar growth target in its guidebook: 
“100% of my students will progress at least one level 
on the FitnessGram during the fall semester.” New 
Jersey (pages 12-14) offers this example of a mastery 
target: “At least 70% of my students will attain a score 
of 80% on the end of course assessment’ Vertically 
aligned courses, such as mathematics courses through 
algebra, typically demand growth targets while 
single-experience courses, such as geometry, lend 
themselves more to mastery targets. 


However, it is important to remember that SLOs 

are not intended to be psychometrically grounded 
growth measures. As the Center for Assessment 
clarifies, “The role of student growth is embedded 
within the process of establishing performance targets 
for groups of students depending on some rough 
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sense of where they start, rather than in the technical 
measurement of change in student performance’ In 
other words, a growth target should not be simply a 
post-test score minus a pre-test score, unless the pre- 
and post-tests are vertically aligned. Educators should 
set targets for a summative assessment based on the 
review of students’ baseline data or information about 
students’ prior knowledge and skills at the start of the 
instructional period. 


The Center for Assessment offers guidance on using 
multiple sources of baseline data to help set targets, 
some of which is summarized in its SLO Toolkit. To 
assess the baseline, educators can examine data from 
State assessments, previous core content classes, 
student work samples or some combination of the 
three. Educators can also conduct their own student 
survey or classroom-based assessment to identify 
students’ starting knowledge. “Once the data have 
been collected, teachers should examine and interpret 
the available data in order to form a comprehensive 
picture of the students in the class. When multiple 
data sources are used and show similar areas of 
student strengths and weaknesses, teachers can be 
more confident in the starting points and the targets 
established” 


* Marion, S.F., DePascale, C., Domaleski, C., Gong, B., and Diaz-Bilello, 
E. (2012). Considerations for Analyzing Educators’ Contributions 
to Student Learning in Non-Tested Subjects and Grades with a 
Focus on Student Learning Objectives. 


° Center for Assessment. (2013). Using Baseline Data and Informa- 
tion to Set SLO Targets: A part of the SLO Toolkit. 
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States increasingly recommend that SLO developers 
set differentiated learning targets based on their 
students’ starting points instead of setting a 

uniform target for all students. Indiana (page 22) 

asks educators to establish targets that are tiered 

or differentiated based on different groupings of 
students’ starting points. The process includes setting 
a “mastery” target for a subject (for example, scoring 

85 points out of 100 on an end-of-course assessment) 
and asking evaluators and teachers to collect baseline 
data, assess student starting points and place them 
into preparedness categories (low, medium and high). 
Then, with guidance based on the number of students 
in each preparedness level, the teacher works with 

her evaluator to determine the overall percentage 

of students expected to perform at the mastery 

level. For example, an excerpt of the guidance from 
Indiana (page 19) reads, “To be considered Highly 
Effective, all students in the high and medium levels of 
preparedness and most of the students in the low level 
achieve content mastery.’ Rhode Island (page 11) and 
Arizona (page 26-36) also recommends tiered targets 
when student knowledge of a course's subject matter 
varies across students. 


Component 3: Communicating with Teachers and Principals 


Ensuring that teachers and principals understand SLOs 
can be as challenging as executing the SLO process. 
Just as SLO procedures composed of checklists and 
calculations require time and focus, conveying the 
rationale behind high-quality SLOs and describing 
their nuts and bolts requires ongoing, thoughtful 
communication between States and educators. States 
that communicate well make clear value propositions 
for different audiences, support their claims with facts 
and resources, and use accessible, varied formats to 
deliver these messages. 


This component of the toolkit aims to help States 
reflect on the quality of their communications 

content, format and delivery to increase educator 
understanding of SLO purpose and process. It does not 
contain an exhaustive inventory of communications 
strategies, but rather an organized compilation of best 
practices. 


The first section shows how States can create 
compelling content by creating an inventory of 
their value propositions, facts and resources. 

The second section offers examples of how States 
are using a variety of formats to connect with 
educators. 

The third and final section describes two delivery 


mechanisms through which States can disseminate 
their communications pieces. 


The RSN has developed a series of tools that 
correspond to each of these three sections and 

has compiled these tools in a Communications 
Workbook. States can use this workbook to adapt the 
best practices described below for their own use. The 
workbook contains five tools: 


1. Making the Value Proposition Worksheet 
. Cataloging Facts Worksheet 

. Organizing Resources Worksheet 

. Two-Page Summary Template 

. FAQs Example 
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Content: Making the Value 
Proposition, Cataloguing Facts and 
Organizing Resources 


States can deliver clear, targeted messages when 
they have internally clarified value propositions for 
different audiences and can draw upon a complete 
and structured inventory of facts and resources. The 
following section showcases effective ways States are 
presenting this information. 


Making the Value Proposition 


A first step in creating or refining a communications 
strategy is crafting clear messages to help educators 
understand the rationale for SLOs. No matter which 
audience they are targeting, States communications 
pieces describe the impact that SLOs can have on 
student learning. For example, New Jersey (page 4) 
describes how SLOs indicate when and how to adjust 
instruction to meet student needs. The State also 
highlights that SLOs help teachers and evaluators 
understand student academic strengths and 
weaknesses. States can also connect with different 
stakeholders by offering tailored value propositions, 
usually by listing the ways SLOs advance teacher and 
principal practice and promote learning across school 
districts. For example, Arizona (page 2) targets 
teachers when it emphasizes that SLOs provide an 
opportunity for teachers to take ownership of a 
portion of their evaluation, and Ohio (page 6) targets 
principals in describing how SLOs focus instruction on 
specific school goals and promote collaboration 
among teachers across classrooms and grade levels. 


States can also help teachers and principals 
understand the rationale behind SLOs by describing 
how SLOs are different than previous processes for 
measuring student growth, especially for teachers of 
non-tested grades and subjects. Presenting contrasts 
in table form can drive home the differences or show 
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that SLOs are a logical continuation of what educators 
already consider best instructional practice. New 
Jersey adopts this practice to communicate about its 
state evaluation framework (Figure 6). 


States can use the first Communications Workbook 
tool, “Making the Value Proposition Worksheet" to 
record and articulate these value propositions for 
appropriate audiences. 


Cataloguing Facts 


States can supplement value propositions that explain 
the why behind SLOs by explaining how teachers and 
administrators can implement them. Best practice 
includes clearly defining whatan SLO is, who SLOs 
affect, and when key changes take place. Unlike 

value propositions, which States often tailor to a 
particular audience, States can avoid confusion and 
ensure consistency by writing facts down once and 
reusing them as needed. States can use the second 
Communications Workbook tool, “Cataloging Facts 
Worksheet’ to identify key facts about the State 

SLO process so that they can communicate them 

to stakeholders. The following subsections show 
potential ways of communicating these basic facts. 


Explaining What an SLO Is 


All States communicate the details of their SLO 
frameworks, although in different ways. The following 
compilation of SLO facts combines content common 
to many State communications pieces and offers one 
potential sequence for explaining SLO basics. 


1. Provide a definition 
States do not all define SLOs in the same way, but 
many States use similar language that describe 
SLOs as goals or targets that are specific and 
measureable, based on available prior student 
learning data, aligned with standards, and based 
on student progress and achievement. Since SLO 
frameworks tend to include nuanced language 
that varies by State, States can avoid confusion 
by stating what they mean by an “SLO” up front. 
For example, New York (page 6) defines SLOs as 
“academic goals for groups of students that are 
aligned to State standards and can be tracked using 
objective measures.’ Ohio (page 5) describes an 
SLO as a“measure of a teacher's impact on student 
learning within a given interval of instruction. 
An SLO is a measurable, long-term academic 


Figure 6. New Jersey's State Evaluation Framework 


State Evaluation Framework 
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Perfunctory In-depth 


Measured by observation only Multiple measures 


One observer Multiple observers 


No connection to 


student growth Student growth counts 


Training not required Comprehensive training required 


Disconnected from 
professional development 


Will provide for more targeted 
professional development 


Minimal educator involvement Educator-driven 


*This is reflective of current minimum requirements by state, and may not reflect practice in all districts. 
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goal informed by available data that a teacher or down the SLO process into discrete steps as Ohio 
teacher team sets at the beginning of the year for has done in Figure 7 can help educators understand 
all students or for subgroups of students” Austin that the process is organized and manageable. 
(page 2) uses yet another definition: “SLOs are 
targets of individual student growth that teachers 
set at the start of the course and strive to achieve 
by the end. These targets are in specific areas 
within State or national standards that have been 
identified as a high need based on a thorough 
review of available data.’ 


3. Explain complex elements 
Most States explain the most challenging elements 
of SLOs. Teachers and administrators commonly 
have the most difficult time understanding 
how to develop learning content, set growth or 
achievement targets and identify appropriate 
assessments. Visuals like those from Indiana 


. Describe the SLO process (pages 8-9) and Arizona (pages 26 and 32) can 
Several States, including Arizona (page 12), New help educators understand these more nuanced 
Jersey (page 10) and Ohio (page 8), use visuals to elements (Figure 8). 


help teachers understand the SLO process. Breaking 


Figure 7. Ohio SLO Process 


1: Gather and review available data 


Figure 8. Arizona Assessments 


Quality Assessments for SLOs 
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4. Demonstrate how SLOs fit into the State's them in isolation, can help reinforce for teachers 
educator evaluation system that SLOs are about improving student learning, not 
States can help audiences understand the just securing a growth measure. 


important part SLOs play in a broad strategy to Explaining Who SOs Atectand How 
improve student achievement and educator quality 


by visually connecting SLOs to the State’s educator States can clear up confusion among teachers and 
evaluation system. Ohio (page 4), Maryland administrators by indicating to which teachers SLOs 
(page 3) and New Jersey use visuals to help apply and how the new system will impact them. 
explain the role that SLOs and other measures of Figure 10 showcases how Indiana (page 4) classifies its 
teacher effectiveness play in an educator's overall teachers into three groups, provides an example of the 
performance evaluation (Figure 9). Coupling types of teachers who fall within each and then uses 
these visuals with information about professional pie graphs to illustrate the SLO contribution to their 
development and support, rather than presenting overall evaluation rating. 


Figure 9. The Role of SLOs in New Jersey's Educator Evaluation System 


2013-2014 Weights: 2013 - 2013 Weights: 
Tested Grades and Subjects Non-Tested Grades and Subjects 
Teachers in Grades 4-8, Teachers Outside of Grades 4-8, 
Language Arts Literacy and Mathematics Language Arts Literacy and Mathematics 
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Figure 10. How SLOs Affect Indiana Teachers 
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Explaining the Timeline for Changes, 
Teacher Response 


States can also assuage educators’ concerns by making 
clear when teachers need to complete certain parts of 
the SLO process. Some States, including Indiana 

(page 6), offer visuals that chart milestones for 
implementing educator evaluation systems and their 
supports (Figure 11). 


States can also help educators understand how SLOs 
fit into the big picture by developing similar, SLO- 


may count in personnel decisions. As shown in Figure 
12, Austin (page 6) provides educators with a detailed 
timeline so that educators can integrate the SLO 
process into their calendars. 


Organizing Resources 


Most States implementing SLOs have produced a 
collection of in-depth guidance documents and 
helpful templates, forms and other process tools to 
help educators understand and use SLOs in their 
work. By pointing audiences to a central, well- 


specific timelines that include, for example, when SLOs organized pool of resources, often a Website, States 


Figure 11. Timeline for Indiana’s RISE Design and Implementation 


Pilot and Refine 


RISE 


Figure 12. Austin School District Timeline 


SLO TIMELINE 2013-2014 


The following is a suggested timeline for year-long SLOs. Deadlines are labeled and 
highlighted. When different, dates and deadlines for semester SLOs can be found in the 
informational box at the bottom of the page. Details regarding each component can 
be found by clicking on the links, or choosing the corresponding section in the TOC. 


AUG — SEPT (1 Complete Needs Assessment 

AUG — SEPT Determine Team or Individual and student group 

SEPT i) Align to standards and create Learning Objective 

SEPT - OCT (1 Develop/choose Assessment and get approval 

SEPT - OCT Administer, Analyze and document Pre-Assessment 
Results 

SEPT - OCT Create Growth Target 

OCTOBER25 [1] Deadline: Complete and submit both SLOs for approval 
in online Database 

SEPT - APR 1 Monitor student progress toward Learning Objective 

NOV -JAN (Complete Revision Requests from principal and/or 
REACH team 

JANUARY 20 (1) Key Date: New Student Enrollment cut-off 

JAN 6 - FEB 3 (Request your own Revision (class or student changes) 

FEB 14 0 Allpre-assessments must be given 

MAR31-MAY9 [1] Administer Post-Assessments (semester deadlines below) 

MAY 16 (© Deadline: Complete Final Submission Form 
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can ensure that intended audiences can access these 
resources. For example, Louisiana provides four lists 
of hyperlinked resources that correspond to four 
questions: “What should students know?" “How will | 
measure success?”, “What are they able to do now?” 
and “How will | monitor progress?” States can use the 
third Communications Workbook tool, “Organizing 
Resources Worksheet” to gather and group resources. 


Format: Developing Effective 
Communications Tools 


States can help educators understand the purpose 
and process complexities of SLOs by packaging their 
messages in a variety of formats. The Communications 
Workbook provides templates to create popular 


formats: a two-page summary, talking points and FAQs. 


States can use the content they develop in the first 
three Communications Workbook tools to populate 
these templates and make the examples their own. 


Two-Page Summary 


Many States have created helpful and detailed 
guidance manuals, but fewer have produced shorter, 
more digestible pieces that provide teachers with 
essential information at-a-glance. New Jersey 

and Indiana have synthesized information in their 
guidebooks into two-page briefs with succinct 
sections on essential topics. New Jersey has also 
produced a quick-start guide that explains five basic 
steps in the SLO process. Each tool provides links to 
the States’ full guidebooks where educators can find 
more detailed information. States can use the fourth 
Communications Workbook tool, the “Two-Page 
Summary Template’ to adapt content they developed 
in the first three tools to create a custom brief. The 
template contains four basic sections: key takeaways, a 
value proposition, SLO facts and additional resources. 


FAQs 


States can create FAQs as a powerful, easy-to-develop 
way to addresses misinformation about SLOs. FAQs 
often communicate information from a teacher's point 
of view, which indicates to educators that SEAs are 
listening and addressing their SLO concerns. States can 
also easily expand them as educators raise additional 
questions over time. Grouping questions by category 
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can help audiences quickly locate answers to their 
questions. For example, Ohio's SLO Web page groups 
by topic nearly 40 frequently asked questions about 
SLOs, which include facts about implementation 

as well as each element of Ohio's SLO template. 
Similarly, New Jersey and Rhode Island's Web pages 
group questions by topic. Other helpful FAQ lists 
include those found in Austin and Indiana's (page 
12-14) SLO guidance manuals. States can browse the 
Communications Workbook’s “FAQs Example" for 
questions that they can use in or adapt for their own 
FAQ documents. 


Summary PowerPoint Presentation 


Several States, including Arizona, New Jersey and 
New York have developed PowerPoint presentations 
that provide overviews of their SLO frameworks. States 
can use these slides to communicate greater detail 
than possible in a two-page summary and in a more 
engaging format than a guidance manual. States 

can use the content they develop in the first three 
Communications Workbook tools to create or revise 
existing slides that district leaders, principals and 
teachers can deliver as part of a“meeting in a box.’ 


Videos 


Rhode Island has produced a short video that 
features educators reflecting on their experiences 
implementing SLOs in their classrooms and schools. 
In their own words, they describe challenges, reflect 
on lessons learned and articulate what they see as the 
benefits of the SLO process. 


Delivery: Reaching Target 
Audiences 


“Meeting in a Box" 


Research by The Winston Group and others confirms 
that principals are by far the most credible messenger 
to teachers about educator evaluation issues, far more 
so than central offices or a State agency. Rhode Island 
and Indiana have developed “meetings in a box" that 
contain tools and resources to support principals 

and teacher ambassadors who understand SLOs and 
can communicate about them clearly and positively. 
States can combine the two-page summary, talking 


points and FAQs developed in the Communications 
Workbook with a PowerPoint presentation to create a 
ready-made package of materials organized together 
in a virtual “box,’ which can be a list of links on a 

Web page, a single PDF document or a “zip” file that 
contains each piece. Rhode Island has found that 
providing principals and teacher leaders with training 
on how to use the “meeting in a box" materials is a best 
practice. 


Organizing Resources on the Web 


Streamlining, grouping by type and providing dates 
for all resources are Website basics. Many States and 
districts, such as Austin, Denver, Georgia, New Jersey, 
New York and Ohio, organize SLO resources on a 
single Web page, which signals to educators that 
they can retrieve the most recent information and 
tools within a few minutes. Often the most accessible 
Web pages group SLO resources into categories that 
reflect how users search for information. For example, 
New Jersey groups information into categories 

such as “How-to Documents” and “Resources and 
Presentations” so users can easily find what they are 
looking for. 


States can prevent the spread of misinformation online 
by indicating when resources are posted. For example, 
Rhode Island's Educator Evaluation home page has 

a prominent ‘Announcements’ section that lists the 
most recent posts to the site so that audiences can 
stay up to date without having to browse through 
multiple Web pages. The State also marks updated 
documents with the word “NEW!!"in bold, red letters 
so that educators always have easy access to the latest 
information. 


States can also use their Websites to make large 
amounts of information more digestible. For example, 
many States communicate valuable information about 
SLOs via comprehensive guidance manuals. However, 
such formats can be overwhelming and make it 


difficult for practitioners to quickly find the information 
they need. Maryland has made its 207-page Teacher 
and Principal Evaluation Guidebook easier to 
comprehend by breaking it into chapter-sized PDF 
chunks that its audiences can browse in outline form 
for context and then view or print a particular section. 


Moving Beyond Informing Teachers and 
Principals Toward Inspiring Them 


The RSN strives with this toolkit section and its 
companion workbook to help States keep educators 
well-informed about the purpose of and processes 
associated with SLOs. However, engaging educators 
goes beyond keeping them informed. The RSN 
encourages States as they implement SLOs to move 
their communications away from simply informing 
audiences and toward inspiring them to act by 
inviting feedback, input and discussion. The RSN has 
developed a communications framework, the “4l’s” 
Framework, to help States meet this challenge. See 
“Component 5: Ensuring Continuous Improvement” 
in this toolkit for strategies States are using to listen to 
educators implementing SLOs in the field. 


Figure 13. RSN “41's” Framework 
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Component 4: Professional Development for SLO 


Implementation 


SLOs can pose practical challenges but also transform 
school and district cultures. They formalize the strong 
instructional practice of gathering data and identifying 
expected student outcomes for every student. They 
also promote dialogue about student learning among 
teachers and between teachers and school leaders. But 
to achieve these positive results requires substantial 
training and support in all stages of the SLO process— 
from development to scoring—to ensure high-quality 
implementation. States and districts employ an array of 
tactics and tools to train their teachers and evaluators 
to create and implement student learning objectives.° 


In-Person Training 


All States implementing SLOs provide in-person 
training, but States have varied target audiences. Some 
focus their training on district administrators, relying 
on districts to train their teachers and evaluators, while 
some also train trainers who redeliver SLO information 
in schools and districts. 


Ohio deploys a sophisticated train-the-trainer 
approach. SLO trainers deliver State-created modules 
after they receive several days of preparatory 
instruction for the role. State or regional delivery 
units around the State train the trainers. Through 

this approach, the Ohio Department of Education 
achieved geographic dispersion of SLO training (the 
State trained hundreds of trainers who in turn trained 
thousands of teachers and principals). While this 
approach was very effective at disseminating critical 
SLO training, Ohio leaders report that there were some 
inherent challenges: expense, workload and version 
control of the training modules. 


© The RSN has produced an Educator Engagement Guide that 
provides a comprehensive way to engage educators and other 
practitioners, including communications and professional devel- 
opment. This framework recognizes a progression of four domains 
of educator engagement that establish the teacher as an active 

| 


subject, the primary actor in a sentence that begins, “| know, 
apply,'"| participate” and “| lead.’ 
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A different train-the-trainer approach is to appoint 
and train a teacher in each school to be an on- 
location resource for teachers as they create their 
SLOs. Both Austin and Houston, for example, pay 
teachers a small stipend to take on such leadership 
roles in their schools. These SLO champions come 
together several times a year, receive training and 
troubleshoot common challenges. Their districts 
give them facilitation materials to conduct “turnkey” 
trainings in their own buildings. Some States use a 
similar approach to deploying SLO experts in regions. 
Rhode Island trains a cohort of “intermediate service 
providers” who support teachers and administrators 
through the entire evaluation process in a slate of 
districts. 


Maryland gives districts considerable flexibility in the 
design of their SLO systems. Thus the State uses several 
training tactics to support quality SLO implementation. 
The Maryland State Department of Education provides 
periodic training for district administrators on State 
SLO regulations and recommendations, convenes 
content-area leaders to brainstorm SLO approaches 

in their field, and deploys customizable Blackboard 
courses, described on page 25. The Maryland State 
Department of Education also engages a cadre of SLO 
trainers that they send to districts to lead or support 
their basic training for teachers regarding SLOs. 


New Jersey is running a series of workshops that 
started at the end of the 2012-2013 school year. 

The workshops began with an introduction to SLOs 
and a specific focus on assessment development. 

The second in the series began in October 2013 and 
addressed specific implementation needs, approval 
protocols and procedures for writing SLOs. New Jersey 
designed these turnkey workshops so that attendees 
can train their own staff using the exact same materials 
provided by the State, along with the pedagogy and 
activities modeled during the workshop. 


Online Training Resources 
and Modules 


Districts and States have developed a range of online 
training tools as a way to make SLO guidance and 
training available on demand. They designed some 
of them to be used with groups, while others they 
designed for individual use. This section describes 
some examples. 


Webinars 


Some States offer SLO webinars, which are then posted 
online for on-demand access. New York, for example, 
provides narrated training webinars, both in video 

and downloadable PowerPoint slides for different 
audiences, including superintendents, principals 

and teachers. Topics include the basic SLO process 

for teachers and principals; how to develop SLOs for 
specific grades and subjects; SLO results analysis; and 
additional State resources. 


PowerPoint Modules 


States and districts provide PowerPoint-based 

training modules, which can be customized for use 
with groups or viewed as they are by individual 

users. Rhode Island’s PowerPoint training modules 
supplement instructor-led, in-person training for 
evaluators, and they include learning exercises and 
discussion questions. Georgia and New Jersey provide 
training modules in PowerPoint format as well. 


Videos 


Like webinars, videos enliven the didactic, technical 
content of SLO training. New York supplements its 
PowerPoint modules for superintendents and districts 
with short, focused videos. Topics include the SLO 
process for teachers and principals, how to develop 
SLOs for specific grades and subjects and State 
resources. Denver and Austin offer user-friendly, short 


videos that stream directly from district Websites. 
Several States plan to upgrade their PowerPoint 
modules to videos during the 2013-2014 school year. 


Online Courses 


Another approach to providing SLO training online 

is to use online courses. With Blackboard and similar 
online course creation and delivery tools, the content 
can be mixed with exercises and quizzes. The course 
format for SLO training promotes user engagement, 
and the platform generates helpful reports about 
users’ mastery of the material. Maryland has deployed 
a Blackboard SLO training system and reports high 
utilization rates and positive feedback from districts. 
Maryland districts adapt the modules for their context 
and use them to augment their homegrown materials. 
Some districts use them as “make-up” training for 
teachers who miss the official training sessions. 

The State provides continuing education credits for 
teachers who complete modules. 


SLO Examples 


All States and school districts implementing SLOs have 
developed examples for a range of grade levels and 
subject areas. They use these samples in conjunction 
with guidance documents to give teachers and 
evaluators illustrations of SLOs and establish 
expectations about how SLOs should be written. For 
example, New York and Rhode Island have posted 
sample SLOs from the field. These provide a variety 

of approaches. New York's are annotated, indicating 
notes of alignment to best practice. 


To extend the successful development of these 
models, the RSN has created an online library 

of annotated SLOs. The library's annotated SLOs 
are searchable by State or subject. Each includes 
embedded comments about the SLO's strengths or 
suggests revisions that would improve the SLO. 
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Component 5: Ensuring Continuous Improvement 


States and school districts want to improve their SLO 
systems continuously to make the most impact on 
instruction, and to make the SLO process as sensible 
as possible for teachers and evaluators. To do so, States 
can monitor implementation of SLOs and analyze their 
results. While States are working to ensure compliance 
and accountability, they also seek to improve the SLO 
system and its supports for teachers and evaluators. 


Assessing System-Wide Quality 


States implementing SLOs report that resources for 
monitoring implementation are very limited. Some 
States, faced with current resource limitations and 
anticipating even greater constraints once Race to the 
Top funding expires, anticipate that the bulk of the 
responsibility for SLO quality monitoring will lie with 
school districts. This reality raises two questions: what 
forms of monitoring should States prioritize, and what 
requirements or guidance for monitoring can they 
offer their districts? 


States and districts should assess their own context 
before deciding how much time, energy and funds to 
invest in a monitoring and results analysis process. As 
they decide to move forward, they can consider such 
options as those presented below. 


State Approaches to Monitoring 
and Continuous Improvement 


States can use three different approaches to 
monitoring the quality of SLOs and SLO systems 

to improve them. Some States implement these 
approaches themselves while others contract with 
outside evaluators to collect the data and conduct the 
analysis. 


1. Qualitative feedback from implementers 
about SLO tools, rules and frameworks. These 
data can be collected through surveys, focus 
groups, evaluation forms following professional 
development sessions, Website or email comment 
collection and on-going open dialogue with 
teachers and evaluators. Taking action based on this 
feedback will generate credibility for the system. 
Louisiana collected feedback throughout the first 
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year on implementation of COMPASS, their new 
evaluation system, and one outcome of educator 
input is that the State will now provide student 
baseline data to all teachers to support their SLO 
target setting. Rhode Island's focus groups and 
surveys of teachers led them to streamline their SLO 
form and improve their SLO approach for special 
education teachers. 


. Analysis of SLO scores. States can average SLO 


scores by content area, school and district to look 
for large differences that warrant further inquiry. For 
example, SLO score analysis might reveal that the 
scores in a few school districts or in a few schools 
are significantly higher than most districts in the 
State. Such differences may indicate authentic 
teacher potency in those locations, or they may 
signal that those districts or schools are setting 
less rigorous targets than their peers. If the State 
concludes that the variations do not reflect true 
differences in teacher effectiveness, the State can 
provide training, adjust guidance or change the 
process to promote calibration. The SLO results 
can also be cross-tabulated with results on other 
components of the evaluation system—for 
example, SLO scores compared to observation 
scores or SLO scores compared to value-added 
scores. Some differences among indicators may be 
perfectly acceptable, but large variations should 
trigger further inquiry. For example, high SLO scores 
where there are low observation results could 
indicate a lack of SLO rigor or they could indicate a 
poor implementation of the observation rubric— 
either way, the source of the discrepancy needs to 
be addressed. This form of monitoring, SLO score 
analysis, is cost effective and will provide invaluable 
data to guide PD and system adjustments that 
promote rigor and comparability. One national 
expert recommends that States prioritize the 
monitoring of scores over the monitoring of SLO 
quality (see approach #3). 


. SLO quality monitoring. States can collect 


samples of SLOs across districts and content areas 
to review the degree to which they comply with 


State guidance, to gauge the level of quality of the 
assessments upon which the SLOs are based and 
to compare the rigor of the targets that teachers 
set. The accuracy of scoring can be assessed by 
double-scoring sample SLOs and comparing the 
State’s recommended score with the score that the 
evaluator gave. Rhode Island uses this approach. 
States can attach accountability to this review by 
requiring districts that submit noncompliant or 
low-quality SLOs to improve them the following 
year, but they may wish to wait for mature 
implementation to do so. In the meantime, they 
can use the review to drive adjustments in SLO 
rules, tools and training. 


Most States plan, additionally, to provide guidance to 
districts about how they should monitor SLOs. The RSN 
plans to collect such guidance as it is developed and 
post it online. 


District Approaches to Monitoring 
and Continuous Improvement 


Districts can pursue a variety of strategies to monitor 
the quality of SLOs and provide data to improve their 
SLO system, 


1. Qualitative feedback from teachers and 


evaluators about SLO tools, rules and frameworks. 


Where there is significant district flexibility in the 
design and implementation of SLOs, qualitative 
data should be collected from schools for district 
analysis. As with the State, districts can collect data 
through surveys, focus groups, evaluation forms 
following professional development sessions, 
Website or email comment collection and on- 
going open dialogue with teachers and evaluators. 
The most important purpose of collecting this 
qualitative feedback is to drive improvements to 
the SLO system, and authentic feedback loops also 
increase the credibility of SLOs among teachers. 


2. Central SLO approval. School districts in Georgia 
as well as other scattered districts around the 
country approve all SLOs centrally. This approach is 
challenging for large districts due to the quantity 
of SLOs, but where feasible, central SLO approval 
promotes consistency across schools and facilitates 
analysis of SLO scores since the district assures 


comparably rigorous targets across teachers and 
schools. 


. Principal accountability. Principals’ skill and 


diligence in approving and scoring SLOs can be 
included in their evaluation. Factors could include 
timeliness of approval and scoring, quality of 
feedback on SLO approval forms and differentiation 
of SLO scores in the school as an indicator of high- 
quality target setting. 


. SLO content analysis. Districts can analyze and 


compare the objective statements, targets and 
assessments used for SLOs across and within 
schools to identify discrepancies, spot innovation 
and assess comparability of rigor. Content directors, 
such as the mathematics director or the science 
director of the district, can best conduct this 
analysis since they know the curriculum and 
assessments used in the SLOs. When SLOs are 
composed on acommon online platform that can 
generate data about particular fields in the SLO 
form, districts can readily analyze their SLOs. 


. SLO score analysis. The district can analyze SLO 


outcomes through statistical analysis of all SLO 
scores, generating comparisons of SLO outcomes 
by school as well as cross-tabulations of SLO 
outcomes with value-added and observation 
scores. Wide variations would trigger a deeper 
inquiry. Where there is significant evaluator 
discretion in SLO scoring, random audits of SLO 
scores will help identify whether some evaluators 
consistently score SLOs higher or lower than their 
colleagues. Such variation could trigger extra 
scoring training. Random audits also promote 
honesty in scoring, since even the remote 
possibility of the discovery of an inflated score will 
deter cheating. 


. SLO-monitoring committee. Districts can 


establish an SLO-monitoring committee to study 
the quality of a sample of SLOs and advise district 
leadership about implications of the samples for 
the SLO system. The committee might identify, 

for example, fields on the SLO template that are 
most challenging for teachers to implement with 
quality, or their review might generate ideas for SLO 
process adjustments. 
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Apendix: State Resources, by Topic 


Introduction 


SLO Context: Definition, Benefits, Challenges and Policy Choices 


Connections with College- and Career-Ready Standards 
Ohio SLO Template Checklist 
Colorado Superintendent and Principal Integrated Timelines 


Component 1: Providing Guidance, Templates and Tools for 
Developing, Approving, and Scoring SLOs 


Developing SLOs Reviewing SLO Progress 
SLO Templates » Indiana Mid-course Check-in Form 


Rhode Island SLO Template Arizona Mid-course Check-in Form (page 39) 


Denver SLO Template 


ring SLOs 

Ohio SLO Template Sco g 

New York SLO Template Delineating Performance Levels 
Louisiana SLO Template - Rhode Island Performance Levels (pages 19) 
Rhode Island SLO Element Guidance - New York Performance Levels (page 16) 


Louisiana SLO Element Guidance (page 13) Calculating Performance Levels 
Pennsylvania SLO Element Guidance 


Indiana SLO Template and Process (pages 36-46) Standardized Approach 


Rhode Island SLO Template - Ohio Scoring Guidance 
Indiana Scoring Guidance (pages 26-28) 
Approving SLOs - Indiana Scoring Guidance (page 19) 


Louisiana Scoring Guidance (pages 16-17) 


Approval Rubrics 
PP Georgia Scoring Guidance 


Denver SLO Approval Rubric 


Austin SLO Rigor Rubric Flexible Approach 
Rhode Island Scoring Guidance (pages 19-21) 
Approval Checklists - Maryland Scoring Guidance 


Rhode Island SLO Quality Review Tool 

Indiana SLO Approval Form 

Ohio SLO Template Checklist 

Louisiana SLO Element Guidance for Approval 
(page 13) 

RSN SLO Approval Checklist 


Hybrid Approach 
Indiana Scoring Guidance (pages 26-28) 
New York Scoring Guidance (pages 16-19) 
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Component 2: Fostering the Selection of High-Quality 
Assessments and Rigorous Targets 


Strategies for Promoting 
Assessment Quality 


Guidance and Training for Selecting or 
Developing High-Quality Assessments 
Rhode Island Assessment Worksheet 
Georgia SLO Measure Criteria Table and Table of 
Specifications (pages 39, 51-57) 
Ohio Assessment Guidance 
Colorado Assessment Review Tool 
Rhode Island Assessment Literacy Module 


Rhode Island Using Baseline Data to Set Targets 
Guide 


Rhode Island Assessment Toolkit 


Assessment Banks 
Colorado Content Collaboratives Resource Bank 


Identifying Pre-Approved Assessments 
Louisiana List and Rank of SLO Assessments 


New York List of Pre-approved, 3rd-Party 
Assessments 


Setting Rigorous and Appropriate 
Targets 


Target Setting with Baseline Data 
Ohio SLO Guidance Manual (pages 9-10) 
New Jersey SLO Guidance Manual (pages 12-14) 
Indiana SLO Target-Setting Guidance (page 19) 
Rhode Island SLO Target-Setting Guidance (page 11) 
Arizona Target-Setting Guidance (pages 26-36) 


Component 3: Communicating with Teachers and Principals 


RSN SLO Communications Workbook 


Content: Making the Value 
Proposition, Cataloging Facts and 
Organizing Resources 


Making the Value Proposition 
New Jersey SLO Value Proposition (page 4) 
Arizona SLO Value Proposition (page 2) 
Ohio SLO Value Proposition (page 6) 


Cataloging Facts 


Explaining What an SLO Is 
New York SLO Definition (page 6) 
Ohio SLO Definition (page 5) 
Austin SLO Definition (page 2) 
Arizona SLO Process Visual (page 12) 


New Jersey SLO Process Visual (page 5) 

Ohio SLO Process Visual (page 8) 

Indiana SLO Assessment Visuals (pages 8-9) 
Arizona SLO Assessment Visuals (pages 26 and 32) 
Ohio Teacher Evaluation System Overview (page 4) 
Maryland State Teacher Evaluation Model (page 3) 
New Jersey Educator Evaluation System Visual 


Explaining Who SLOs Affect and How 
Indiana Teacher Groups (page 4) 


Explaining When Changes Take Effect and When 
Teachers Need to Take Certain Actions 

Indiana Timeline (page 6) 

Austin Timeline (page 6) 


Organizing Resources 
Louisiana Target-Setting Guide 
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Format: Developing Effective Delivery: Reaching Target 


Communication Pieces Audiences 
Two-Page Summary Organizing Resources on the Web 
New Jersey Two-Page Summary - Austin SLO Web Page 
Indiana Three-Page Summary - Denver SLO Web Page 
New Jersey Quick Start Guide - Georgia SLO Web Page 
FAQs New Jersey SLO Web Page 
New York SLO Web Page 
Cnr Ohio SLO Web Page 
cilia: Aiditeed Rhode Island Educator Evaluation Web Page 
Rhode SEG EAGs Rhode Island SLO Web Page 
Austin FAQs 


Maryland SLO Guidebook Chapters 
Indiana FAQs (page 12-14) 


Moving Beyond Informing Teachers and 


Summary PowerPoint Presentation Principals Toward Inspiring Them 
Arizona Summary - RSN“4I's” Framework 
New Jersey Summary RSN Educator Engagement Guide 
New York Summary 

Videos 


Rhode Island SLO Video 


Component 4: Professional Development for SLO Implementation 


In-Person Training - Georgia PPT Training Module 


New Jersey PPT Training Module 
Ohio Train-the-Trainer Modules 


New Jersey SGO Workshop Presentation Videos 
New Jersey SGO Workshop Participant Packet - New York SLO Process Videos 
Denver SLO Video 
Online Training - Austin SLO Video 


Resources and Modules 
Online Courses 


Webinars - Maryland Blackboard SLO Materials 


New York SLO Webinars 
SLO Examples 


PowerPoint Modules - New York SLO Examples 
Rhode Island PPT Training Modules - Rhode Island SLO Examples 
Rhode Island In-Person Training - RSN Online Library of Annotated SLOs 


Component 5: Ensuring Continuous Improvement 


State Approaches to Monitoring and Continuous Improvement 


Louisiana Educator Feedback 
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